Papers with attention layer
Align before Attend: Aligning Visual and Textual Features for Multimodal Hateful Content Detection (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing approaches to multimodal hateful content detection focus on detecting hate speech from text-based content, but they fail to address modality-specific features. |
| Approach: | They propose a context-aware attention framework for multimodal hateful content detection that integrates an attention layer to meaningfully align the visual and textual features. |
| Outcome: | The proposed framework achieves F1-scores of 69.7% and 70.3% on two hateful meme datasets and shows 2.5% and 3.2% performance improvement over the state-of-the-art systems. |
On Sentence Representations for Propaganda Detection: From Handcrafted Features to Word Embeddings (D19-50)
Copied to clipboard
| Challenge: | The rise of fake and hyperpartisan news on social media and online news outlets calls for improved automatic detection of propaganda in texts. |
| Approach: | They propose to use handcrafted features and learn dense semantic representations to detect propaganda in sentence-level and with random undersampling of the majority class (non-propaganda) |
| Outcome: | The proposed system achieves a ranking of 10 among 25 participants, with 59.5 F1-score. |
How does Attention Affect the Model? (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on the effectiveness of attention in NLP do not consider changes in semantic capability of different components. |
| Approach: | They propose a framework that exploits a convex hull representation of sequence semantics in an n-dimensional Semantic Euclidean Space and defines indicators to capture the impact of attention on sequence semantic. |
| Outcome: | The proposed framework exploits a convex hull representation of sequence semantics in an n-dimensional Semantic Euclidean Space and defines indicators to capture the impact of attention on sequence semantic. |
Adversarial Attention Modeling for Multi-dimensional Emotion Regression (P19-1)
Copied to clipboard
| Challenge: | Empirical evaluation on EMOBANK corpus shows that our approach achieves notable improvements in r-values over the state-of-the-art baselines. |
| Approach: | They propose a neural network-based approach to multi-dimensional emotion regression which automatically rates multiple emotion dimension scores for an input text. |
| Outcome: | The proposed approach achieves notable improvements in r-values on both EMOBANK Reader’s and Writer’s multi-dimensional emotion regression tasks over the state-of-the-art baselines. |
Multi-turn Response Selection using Dialogue Dependency Relations (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for multi-turn response selection ignore the dependencies between the turns. |
| Approach: | They propose a dialogue extraction algorithm to transform a dialog history into threads based on their dependency relations. |
| Outcome: | The proposed model outperforms the state-of-the-art models on DSTC7 and DSTF8* with competitive results on UbuntuV2 . |
Aspect Based Sentiment Analysis with Gated Convolutional Networks (P18-1)
Copied to clipboard
| Challenge: | Aspect-based sentiment analysis can provide more detailed information than general sentiment analysis. |
| Approach: | They propose a model based on convolutional neural networks and gating mechanisms which can selectively output the sentiment features according to the given aspect or entity. |
| Outcome: | The proposed model can selectively output sentiment features according to the given aspect or entity. |
Structured Minimally Supervised Learning for Neural Relation Extraction (N19-1)
Copied to clipboard
| Challenge: | Recent work shows that distant supervision can cause significant label noise when learning from large quantities of unlabeled text. |
| Approach: | They propose a method that combines the benefits of learning representations and structured learning to predict sentence-level relation mentions given only proposition-level supervision from a KB. |
| Outcome: | The proposed approach outperforms a number of baseline approaches while minimizing label noise. |
APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng, Lifu Huang, Xiaojun Quan, Zenglin Xu, Dongfang Liu
| Challenge: | Existing prompt tuning methods only introduce prompts at the input layer, limiting performance and leaving large room for improvement. |
| Approach: | They propose a method that involves tuning a small set of soft prompts for pre-trained language models. |
| Outcome: | The proposed method outperforms state-of-the-art methods with pre-trained models on the SuperGLUE benchmark. |
Finding Educationally Supportive Contexts for Vocabulary Learning with Attention-Based Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Identifying educationally supportive contexts for vocabulary learning is an important problem to solve for designing effective curricula for contextual word learning. |
| Approach: | They evaluate attention-based approaches to find supportive contexts for vocabulary learning scenarios using an existing benchmark dataset. |
| Outcome: | The proposed model outperforms a generic model and a custom model on a major dataset for educational context support prediction. |
Multi-Task Stance Detection with Sentiment and Stance Lexicons (D19-1)
Copied to clipboard
| Challenge: | Recent studies show improvements in stance detection by using attention mechanism or sentiment information. |
| Approach: | They propose a multi-task framework that incorporates attention mechanism and takes sentiment classification as an auxiliary task. |
| Outcome: | The proposed model outperforms state-of-the-art deep learning methods on the SemEval-2016 dataset. |
Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have found that low-rank pre-training often compromises effectiveness. |
| Approach: | They propose to apply low-dimensional module only to the attention layer to improve both effectiveness and efficiency. |
| Outcome: | The proposed model saves 12.4% time while improving test perplexity and on downstream tasks compared with vanilla Transformer. |
On the Expressivity Role of LayerNorm in Transformers’ Attention (2023.findings-acl)
Copied to clipboard
| Challenge: | Layer Normalization (LayerNorm) is an inherent component in all Transformer-based models. |
| Approach: | They propose to use LayerNorm to normalize the activations during the forward pass and their gradients during the backward pass. |
| Outcome: | The proposed model is able to express the multi-head attention layer that follows it in a d-1 space and scales to the same norm of d. |